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(57) A full-scene storage portion (18) stores full- 
scene BIFS data and if a BIFS command other than 
Scene Replace is received from an encoder (1 2) a full- 
scene composition portion (16) composites the full- 
scene BIFS data stored in the full-scene storage portion 
(18) and the BIFS command together to generate new 



full-scene BIFS data and update the full-scene BIFS da- 
ta stored in the full-scene storage portion (18) and pe- 
riodically or if a reception system issues a transmission 
request the full-scene BIFS data stored in the full-scene 
storage portion (18) is transmitted to the reception sys- 
tem. Thus a scene can be displayed with high quality. 
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Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 



[0001] The present invention relates to transmission 
systems, reception systems, and transmission and re- 
ception systems transmitting and/or receiving multime- 
dia information. 

Description of the Background Art 

[0002] In the moving pictures experts group (MPEG)- 
4 system, scene descriptions described in a scene de- 
scription language binary format for scenes (BIFS) can 
be used to composite speeches, animated Images, 
graphics and other similar various objects and display 
the same as a scene. 

[0003] A scene description is described with a tree 
structure referred to as a scene graph, having each 
node with an object associated therewith. The scene 
graph can be dynamically updated by a BIFS command 
described in the scene description language BIFS. 
Hereinafter, data described in the BIFS including a BIFS 
command changing a scene graph will be referred to as 
BIFS data. 

[0004] A scene description and an object serving as 
a constituent of the same description are associated 
with each other by an object descriptor corresponding 
to control information. 

[0005] Object data, BIFS data and an object descrip- 
tor are each delivered from a transmitting side to a re- 
ceiving side in a stream in the form of a packet, referred 
to as an SL packet, one at a time. 
[0006] An object packet and a BIFS packet may be 
multiplexed together and thus transmitted in a single 
stream or they may be transmitted in different streams 
via different routes and protocols. 
[0007] Typically the Bl FS packet is generated non-pe- 
riodically for example when scenes are switched, 
whereas typically the object packet is periodically gen- 
erated. 

[0008] Furthermore, the BIFS data, as well as a pro- 
gram file, is not allowed to have data missing and It must 
be received by a receiving side in its complete condition, 
whereas the object data such as a speech, an animated 
image and the like may have data more or less missing 
although it should be transmitted in real time. 
[0009] As such, if the BIFS data and the object data 
are multiplexed together and thus transmitted in a single 
stream, transmitting a speech, an animated image and 
the like in real time is typically prioritized and they are 
thus transmitted according to a protocol which does not 
perform retransmission, e.g., user data protocol (UDP). 
[0010] If a protocol does not provide retransmission, 
the BIFS packet can be lost. Inparticular, if BIFSpackets 
are communicated by wireless they can often, succes- 



sively be lost. 

[0011] If a packet is transmitted according to a proto- 
col providing retransmission (e.g., transmission control 
protocol (TCP)) it can be prevented from being missing, 
5 although reproduction fails to maintain the time structure 
of a speech, an animated image and the like and it can- 
not be provided smoothly or naturally. 
[0012] Accordingly, separating and transmitting BIFS 
data and object data according to different protocols, 
10 may also be considered. For example, BIFS data is 
transmitted according to a protocol including retrans- 
mission whereas object data is transmitted according to 
a protocol excluding retransmission. 
[0013] In this example, however, BIFS data can arrive 
15 later than the corresponding object data by the period 
of time taken for the BIFS data to be retransmitted. 
[0014] In other words, the transmission system can- 
not be free of the possibility that BIFS data will be lost 
or delayed. 

20 [0015] Missing or delayed BIFS data results in the fol- 
lowing disadvantages: 

[0016] Firstly, if BIFS data delays, the corresponding 
speech, animated image or the like may not be dis- 
played or it may be delayed if it is displayed. 
25 [0017] Secondly, in switching a scene if a contradic- 
tion is introduced into a BIFS command then data can 
be displayed abnormally. For example, if an animated 
image is being displayed with a Japanese speech cor- 
responding thereto, switching the Japanese speech to 
30 an English speech requires a command deleting the 
Japanese speech from the scene and a command add- 
ing the English speech to the scene. However, if the 
command deleting the Japanese speech from the scene 
is somehow missing (or delayed) the animated image is 
35 displayed with the Japanese and the English speeches 
simultaneously displayed. 

[0018] Furthermore in BIFS data there are a BIFS for 
the entirety of a scene and a BIFS command and if the 
BIFS command is lost an abnormal output will continue 
40 until the BiFS for the entirety of a subsequent scene is 
received. In particular if a BIFS for the entirety of a 
scene is followed by using a large number of BIFS com- 
mands then recovery is not likely to be achieved. 

45 SUMMARY OF THE INVENTION 

[0019] The present invention therefore contemplates 
a transmission system, reception system and transmis- 
sion and reception system allowing a scene to be dis- 
50 played with high quality. 

[0020] In the present invention the reception system 
uses scene description information describing a scene 
transmitted from a transmission system and object data 
constituting the scene, to display the scene, and it in- 
55 eludes a composition process portion using the scene 
description information to composite a corresponding 
the object data to display the scene, and a display con- 
trol portion controlling the composition process portion 
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to resolve an error in the scene displayed that is attrib- 
uted to non-arrival of scene description information pro- 
vided to describe the scene. 

[0021 ] Preferably in controlling the composition proc- 
ess portion to resolve the error the display control por- 
tion supplies the composition process portion with the 
scene description information for resolving the error and 
controls the composition process portion to use the 
scene description information for resolving the error to 
resolve the error. 

[0022] More preferably the display control portion in- 
cludes a determination portion determining arrival/non- 
arrival of the scene description information, and a con- 
trol portion supplying the composition process portion 
with default scene description information and control- 
ling the composition process portion to use the default 
scene description Information to execute the composi- 
tion when the determination portion determines that the 
scene description information has not yet arrived. 
[0023] Still more preferably the control portion in- 
cludes a memory storing the default scene description 
information read according to a determination made by 
the determination portion. 

[0024] Furthermore still more preferably, the trans- 
mission system also transmits scene control information 
associating the object data and the scene description 
information with each other, the composition process 
portion includes a compositor using the scene control 
information to composite corresponding object data, 
and the control portion also includes a modification por- 
tion operative in response to a determination of the de- 
termination portion to use the default scene description 
information to modify corresponding scene control infor- 
mation for output to the composition process portion. 
[0025] In particular the memory stores scene descrip- 
tion information previously received. 
[0026] Preferably, the scene description information 
includes full-scene description information for describ- 
ing a full scene and a scene description command par- 
tially changing the scene, and the display control portion 
includes a determination portion determining whether 
received scene description information has an abnor- 
mality, and a transmission requesting portion requesting 
the transmission system to transmit full-scene descrip- 
tion information when the determination portion deter- 
mines that received scene description information has 
an abnormality. 

[0027] Preferably when the scene description infor- 
mation does not have a successive sequence number 
the determination portion determines that there is an ab- 
normality. 

[0028] Preferably the sequence number is a se- 
quence number of a sink-layer packet. 
[0029] Preferably the display control portion includes 
a check portion checking whether there is a contradic- 
tion between received scene description information 
and a current display, and a control portion eliminating 
a contradiction in a display occurring in the composition 



process portion when the check portion determines that 
the contradiction has been introduced. 
[0030] Preferably the check portion includes a pass/ 
fail determination database storing information referred 
5 to to make the determination. 

[0031 ] Preferably the check portion refers to the infor- 
mation referred to to make the determination and if it is 
found therefrom that scene description information to be 
received is missing or delayed then the check portion 
10 determines that the contradiction has been introduced. 
[0032] Preferably the control portion is operative in re- 
sponse to an output of the check portion to modify the 
current display. 

[0033] Preferably the control portion is operative in re- 
's sponse to an output of the check portion to modify scene 
description information being used for the current dis- 
play. 

[0034] Furthermore in the present invention the trans- 
mission system transmits scene description information 
so including full-scene description information for describ- 
ing a full scene or a scene description command for par- 
tially changing the scene, and an object data constitut- 
ing the scene, and it includes: a scene description infor- 
mation encoding portion encoding the scene description 
25 information; an object encoding portion encoding the 
object data; a storage portion storing the full-scene de- 
scription information; a composition portion operative in 
response to a predetermined scene description com- 
mand received from the scene description information 
30 encoding portion, to composite full-scene description in- 
formation stored in the storage portion and the prede- 
termined scene description command together to gen- 
erate and use new full-scene description information to 
overwrite and thus update the full-scene description in- 
35 formation in the storage portion for storage; a multiplex- 
ing portion multiplexing together scene description in- 
formation received from the scene description informa- 
tion encoding portion or full-scene description informa- 
tion received from the storage portion and object data 
received from the object encoding portion; and a switch 
portion switching an input for the multiplexing portion be- 
tween the scene description information encoding por- 
tion and the storage portion. 

[0035] Preferably the composition portion provides 
45 the composition only when a command other than a 
command Instructing that a scene be entirely replaced 
is received. 

[0036] Preferably the transmission system periodical- 
ly transmits to a reception system the full-scene descrip- 
50 tion information stored in the storage portion. 

[0037] Preferably the transmission system receives 
from a reception system receiving the object data and 
the scene description information a request made to 
transmit the full-scene description information, and re- 
55 sponsively transmits the full-scene description informa- 
tion to the reception system. 

[0038] Furthermore in the present invention the trans- 
mission and reception system includes a transmission 
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system transmitting scene description information for 
describing a scene and object data constituting the 
scene, and a reception system receiving the scene de- 
scription information and the object data from the trans- 
mission system and using the scene description infor- 5 
mation to composite the object data to display the 
scene. The transmission system includes: a scene de- 
scription information encoding portion encoding the 
scene description information; an object encoding por- 
tion encoding the object data; a storage portion storing 10 
full-scene description information for describing the 
scene entirely; a composition portion operative in re- 
sponse to a predetermined scene description command 
received from the scene description information encod- 
ing portion, to composite full-scene description informa- '5 
tion stored in the storage portion and the predetermined 
scene description command together to generate and 
use new full-scene description information to overwrite 
and thus update the full-scene description information 
in the storage portion for storage; a multiplexing portion 20 
multiplexing together scene description information re- 
ceived from the scene description information encoding 
portion or full-scene description information received 
from the storage portion and object data received from 
the object encoding portion; and a switch portion switch- 25 
ing an input for the multiplexing portion between the 
scene description information encoding portion and the 
storage portion. The reception system includes a deter- 
mination portion determining whether received scene 
description information has an abnormality, and a trans- 30 
mission requesting portion requesting the transmission 
system to transmit full-scene description information 
when the determination portion determines that re- 
ceived scene description information has an abnormal- 

ity. 

[0039] The foregoing and other objects, features, as- 
pects and advantages of the present invention will be- 
come more apparent from the following detailed de- 
scription of the present invention when taken in conjunc- 
tion with the accompanying drawings. 40 

BRIEF DESCRIPTION OF THE DRAWINGS 



[0040] In the drawings: 



45 



Fig. 1 Is a block diagram showing a transmission 
and reception systems in a first embodiment of the 
present invention; 

Fig. 2 is a schematic block diagram showing a con- 
figuration of the transmission system in the first em- so 
bodiment; 

Figs. 3 and 4 are views each for illustrating a func- 
tion of the full-scene composition portion shown in 
Fig. 2; 

Fig. 5 is a block diagram schematically showing a 55 
configuration of the reception system in the first em- 
bodiment; 

Fig. 6 is a conceptual portion for illustrating a func- 



tion of an object descriptor; 
Figs. 7 and 8 are first and second flow charts, re- 
spectively, for illustrating an operation in the Fig. 1 
transmission system; 

Figs. 9 and 10 are first and second flow charts, re- 
spectively, for illustrating an operation of the Fig. 1 
reception system; 

Fig. 11 is a conceptual view showing a relationship 
between reception and transmission systems in a 
second embodiment of the present invention; 
Fig. 12 is a block diagram schematically showing a 
configuration of the reception system in the second 
embodiment; 

Figs. 13A and 13B are conceptual views for illus- 
trating a function of the reception system shown in 
Fig. 11; 

Figs. 14 and 15 are each a flow chart of acquisition 
of session data In the MPEG-4 system; 
Fig. 16 is a flow chart representing a portion of a 
process to acquire session data in the Fig. 11 re- 
ception system; 

Fig. 17 is a flow chart of a process provided to de- 
termine whether BIFS data has arrived in the Fig. 
11 reception system; 

Fig. 1 8 is a conceptual view representing a relation- 
ship between reception and transmission systems 
in a third embodiment of the present invention; 
Figs. 19A and 19B are conceptual views for illus- 
trating a function of the reception system shown in 
Fig. 18; 

Fig. 20 is a flow chart representing a portion of a 
process to acquire session data in the Fig. 1 8 re- 
ception system; 

Fig. 21 is a flow chart of a BIFS extraction process 
in the Fig. 18 reception system; 
Figs. 22 and 23 are each a block diagram schemat- 
ically showing a configuration of a main portion of 
the reception system in the third embodiment; 
Fig. 24 is a conceptual view representing a relation- 
ship between reception and transmission systems 
in a fourth embodiment of the present invention; 
Fig. 25 is a flow chart for illustrating a process pro- 
vided to check BIFS data in the reception system 
shown in Fig. 24; 

Fig. 26 is a block diagram schematically showing a 
configuration of a main portion of the reception sys- 
tem in the fourth embodiment; 
Figs. 27 and 28 are block diagrams each schemat- 
ically showing another configuration of the main 
portion of the reception system in the fourth embod- 
iment; and 

Fig. 29 is a conceptual view representing a function 
of the reception system in the fourth embodiment. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0041] The embodiments of the present invention will 
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now be described more specifically with reference to the 
drawings. In the figures, like components are denoted 
by like reference characters and will not be described 
repeatedly. 



First Embodiment 

[0042] Fig. 1 is a block diagram schematically show- 
ing a system transmitting and receiving multimedia in- 
formation in the MPEG-4 system. As shown in Fig. 1 , a 
transmission and reception system 100 includes a 
transmission system 1 0 and a reception system 20. 
[0043] Transmission system 10 encodes, in a prede- 
termined system, object data such as a speech, an an- 
imated image, a still picture and computer graphics 
(CG). BIFS data provided for compositing object data to 
describe a scene, and an object descriptor associating 
the BIFS data and the object data with each other, and 
transmits them to reception system 20. 
[0044] Reception system 20 receives the object data 
from transmission system 1 0 and composites the object 
data by using the BIFS data and the object descriptor 
and displays each scene. 

[0045] With reference to Fig. 2, transmission system 
10 includes an encoder group 10A, a multiplexer 14, a 
transmission and reception portion 15, a full-scene com- 
position portion 16, a full-scene storage portion 18 and 
an output switch portion 19. Encoder group 10A is 
formed of encoders 10a, 10b, .... 10n and encoders 11 
and 12. 

[0046] Encoders 10a, 10b, 10n are provided for 
object data, such as a speech, an animated image, a 
still picture, CG and the like, in a one to one correspond- 
ence, and each encode different object data. For exam- 
ple, encoder 10a encodes a speech, encoder 10b en- 
codes an animated image, and encoder 10n encodes 
the other object data. Encoder 1 1 encodes an object de- 
scriptor OD and encoder 12 encodes BIFS data. 
[0047] Multiplexer 14 multiplexes together in a multi- 
plexing stream format the object data encoded by and 
received from encoder 10a-10n, object descriptor OD 
encoded by and received from encoder 1 1 and the BIFS 
data encoded by and received from encoder 12 and out- 
puts the same to transmission and reception portion 1 5. 
[0048] Transmission and reception portion 15 trans- 
mits the data multiplexed by multiplexer 1 4 to reception 
system 20 via a public line or, as will be described here- 
inafter, receives a request sent from a transmission sys- 
tem via a public line network to transmit BIFS data and 
outputs the request to output switch portion 19. 
[0049] Full-scene composition portion 1 6 composites 
full-scene BIFS data that is stored in full-scene storage 
portion 18 and a BIFS command received from encoder 
12 together to generate new full-scene BIFS data, up- 
dates the full-scene BIFS data in full-scene storage por- 
tion 18 by using the new, generated full-scene BIFS da- 
ta, and stores the updated BIFS data in full-scene stor- 
age portion 18. 



[0050] Full-scene composition portion 16 functions, 
as will more specifically be described hereinafter. With 
reference to Fig. 3, for example if a scene description 
currently being displayed is represented by a scene 
5 graph formed of a node A1 (a square frame), a node 
A10 (a news caster (an animated image)) and a node 
11 (a Japanese speech), full-scene storage portion 18 
stores full-scene BIFS data with the Fig. 3 node A1 and 
nodes A10 and All connected thereto. 
io [0051] Then when encoder 12 issues a BIFS com- 
mand instructing that the Japanese speech be discon- 
nected, a node 12 (an English speech) and a BIFS com- 
mand instructing that the English speech be connected, 
full-scene composition portion 16 disconnects node All 
'5 from node A1 adds node A12 and connects node 12 to 
node A1 (see Fig. 4). Thus, full -scene composition por- 
tion 16 composites full-scene BIFS data stored in full- 
scene storage portion 1 8 and a Bl FS command received 
from encoder 12 together and full-scene storage portion 
2 o 18 is overwritten with the new, composited full-scene 
BIFS command. Thus, full-scene storage portion 18 
constantly stores the latest full-scene BIFS data. 
[0052] Full-scenecomposition portion 1 6 provides the 
above-described composition only when it receives 
25 from encoder 1 2 a BIFS command other than Scene Re- 
place, and full-scene composition portion 16 does not 
provide the above-described composition when the 
Scene Replace BIFS command is received from encod- 
er 12, since the Scene Replace BIFS command in- 
30 structs that previous full-scene BIFS data be replaced 
by new full-scene BIFS data and full-scene BIFS data 
having been stored in full-scene storage portion 18 can 
thus not be composited with a BIFS command received 
from encoder 12. Thus, when Scene Replace BIFS 
35 command is received, full-scene composition portion 16 
overwrites full-scene storage portion 18 with full-scene 
BIFS data subsequently received and updates full- 
scene BIFS data previously stored. Note that a BIFS 
command other than Scene Replace includes Insertion, 
*o Deletion, and Replace. 

[0053] Full-scene storage portion 1 8 responds to a re- 
quest received from full-scene composition portion 16 
by outputting stored full-scene BIFS data to full-scene 
composition portion 16 and also overwriting previously 
^5 stored full-scene BIFS data with full-scene BIFS data 
output from full-scene composition portion 16 and thus 
storing the full-scene Bl FS data received from full-scene 
composition portion 16. 

[0054] Output switch portion 1 9 includes terminals X 
50 and Y and in normal operation has terminal X connected 
to receive BIFS data from encoder 12 and output the 
BIFS data to multiplexer 14 and when it receives from 
transmission and reception portion 15 a request for 
transmission it has terminal Y connected to read full- 
55 scene BIFS data from full-scene storage portion 1 8 and 
output the BIFS data to multiplexer 14. Output switch 
portion 1 9 is thus adapted to have a terminal for an input 
to multiplexer 14 that can be switched to prevent simul- 
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taneous transmission of a BIFS command from encoder 
1 2 and full-scene BiFS data from full-scene storage por- 
tion 18. More specifically, a BIFS command and full- 
scene BIFS data are given the same time stamp, e.g., 
a composition time stamp (CTS), a decoding time stamp 
(DTS) or the like, and if they are simultaneously trans- 
mitted, confusion may be introduced on the receiving 
side. 

[0055] With reference to Fig. 5, reception system 20 
includes a transmission and reception portion 24, a de- 
multiplexer 25, a decoding buffer group 30, a decoder 
group 40, a composition memory group 50, a compos- 
iter 60, a BIFS check portion 70, and a retransmission 
request generation portion 80. Decoding buffer group 
30 is formed of decoding buffers 30a-30n and decoding 
buffers 31 and 32. Decoder group 40 is formed of de- 
coders 40a-40n and decoders 41 and 42. Composition 
memory group 50 is formed of composition memories 
50a-50n and composition memories 51 and 52. 
[0056] Transmission and reception portion 24 re- 
ceives object data, BIFS data and object descriptor OD 
from transmission system 10 via a public line network 
and outputs the object data, object descriptor OD and 
the BIFS data to demultiplexer 25. Transmission and re- 
ception portion 24 also receives a retransmission re- 
quest signal from retransmission request generation 
portion 80 and transmits the signal to transmission sys- 
tem 10 via a public line network. 
[0057] Demultiplexer 25 receives object data, object 
descriptor OD and BIFS data from transmission and re- 
ception portion 24 and outputs them to decoding buffers 
30a-30n, decoding buffer 31 and decoding buffer 32, re- 
spectively. 

[0058] Decoding buffers 30a-30n are provided for ob- 
ject data, such as a speech, an animated image, a still 
picture, CG and the like, in a one to one correspond- 
ence, and each temporarily hold different object data. 
For example, decoding buffer 30a temporarily holds a 
speech and outputs it to decoder 40a, decoding buffer 
30b temporarily holds an animated image and outputs 
it to decoder 40b and decoding buffer 30n temporarily 
holds the other object data and outputs it to decoder 
40n. Decoding buffer 31 temporarily holds object de- 
scriptor OD and outputs it to decoder 41 and decoding 
buffer 32 temporarily holds BIFS data and outputs it to 
decoder 42. 

[0059] Decoders 40a-40n are provided for object da- 
ta, such as a speech, an animated image, a still picture, 
CG and the like, in a one to one correspondence, and 
each decode different object data. For example, decod- 
er 40a decodes a speech, decoder 40b decodes an an- 
imated image and decoder40n decodes the other object 
data. Decoder 41 decodes object descriptor OD and de- 
coder 42 decodes BIFS data. Each of decoders 40a-40n 
and 41 and 42, with an access unit AU corresponding 
to an input, outputs a composition unit. 
[0060] Composition memories 50a-50n hold the com- 
position units output from their respective decoders 40a- 



" 40n. Each composition memory corresponds to one of 
object data. For example, composition memory 50a cor- 
responds to a speech, composition memory 50b corre- 
sponds to an animated image, and composition memory 
5 50n corresponds to the other object data. Composition 
memory 51 holds a composition unit output from decod- 
er 41 . Composition memory 52 holds a composition unit 
output from decoder 42. 

[0061] Compositor 60 composites object data re- 
10 ceived from composition memories 50a-50n by using 
object descriptor OD from composition memory 51 and 
BIFS data from composition memory 52, to display a 
scene. 

[0062] Reference will now be made to Fig. 6 to more 
is specifically describe a function of compositor 60. Com- 
positor 60 for example receives BIFS data 60 from com- 
position memory 52, object descriptors 62-64 from com- 
position memory 51 , and an elementary stream 65 from 
composition memories 50a-50n. BIFS data 61 includes 
20 a BIFS for describing three object data corresponding 
to a character 611 , furniture 612 and a background 613 
together configuring one scene. In BIFS data 61 , char- 
acter 612, furniture 612 and background 613 are as- 
signed different object descriptor identifications OD- 
25 |D1 , OD-ID2 and OD-ID3, respectively. 

[0063] Compositor 60 refers to object descriptor iden- 
tifications OD-ID1, OD-ID2 and OD-ID3 assigned to 
character 611, furniture 612 and background 613, re- 
spectively, of BIFS data 61 and identify the object de- 
30 scriptors corresponding to character 611 , furniture 612 
and background 613. More specifically, compositor 60 
detects object descriptor 62 with object descriptor iden- 
tification OD-ID 1 as an object descriptor corresponding 
to character 611. Similarly compositor 60 detects object 
35 descriptor 63 as an object descriptor corresponding to 
furniture 612 and object descriptor 64 as an object de- 
scriptor corresponding to background 613. In each of 
object descriptors 62-64 there exists an elementary- 
stream descriptor ES-D describing information associ- 
40 ating the object descriptor with elementary stream ES. 
Elementary-Stream descriptor ES-D provides an asso- 
ciation with stream 65 corresponding to object data. 
More specifically, elementary-stream descriptor ES-D 
includes an elementary stream identification ES-ID. 
45 [0064] Thus, compositor 60 refers to elementary- 
stream descriptor ES-D11 Included In object descriptor 

62 and further refers to elementary stream identification 
ES-ID 11 included in elementary-stream descriptor ES- 
D 11. Thus compositor 60, referring to elementary 

so stream identification ES-ID11, acquires elementary 
stream ES11 corresponding to character 611 contained 
in stream 65. Similarly, compositor 60 refers to elemen- 
tary stream identification ES-ID21 in object descriptor 

63 to acquire elementary stream ES21 corresponding 
55 to furniture 61 2 and refers to elementary stream identi- 
fication ES-ID31 in object descriptor 64 to acquire ele- 
mentary stream ES31 corresponding to background 
613. Thus, acquired elementary streams ES11, ES21 
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and ES31 are composited together to display a single 
scene. 

[0065] Thus, object descriptor OD associates BIFS 
data with an elementary stream. 
[0066J Again with reference to Fig. 5 BIFS check por- 
tion 70 checks BIFS data, which is based on a compo- 
sition unit output from decoder 42. More specifically, 
BIFS check portion 70 checks BIFS data in a method as 
described hereinafter, according to a sequence number 
of a packet (an SL packet). According to a result ob- 
tained from BIFS check portion 70, retransmission re- 
quest generation portion 80 requests transmission sys- 
tem 10 to retransmit full-scene BIFS data. Retransmis- 
sion request generation portion 80 generates data for 
the retransmission request and outputs the generated 
data for the retransmission request to transmission and 
reception portion 20. 

[0067] The transmission and reception system con- 
figured as described above operates as will be de- 
scribed hereinafter. Initially, the operation of transmis- 
sion system 10 will be described. 
[0066] As has been described previously, encoder 
group 1 0 encodes object data such as a speech, an an- 
imated image and the like, an object descriptor, and 
BIFS data. More specifically, encoders 10a-10n encode 
the object data, encoder 1 1 encodes the object descrip- 
tor and encoder 12 encodes the BIFS data. Then en- 
coders 1 0a-1 On and 1 1 and 1 2 each output the encoded 
data to multiplexer 14. Note that output switch portion 
1 9 normally has terminal X connected and the data from 
encoder 12 is input to multiplexer 14. Furthermore en- 
coder 12 also outputs the encoded data to full-scene 
composition portion 16. 

[0069] Full-scene composition portion 16 operates, 
as represented in Fig. 7. Full-scene composition portion 
16 determines whether BIFS data has been acquired 
from encoder 12 (step S10). Then, full-scene composi- 
tion portion 1 6 repeats the step S1 0 decision until BIFS 
data is acquired from encoder 1 2. When full-scene com- 
position portion 16 determines that BIFS data has been 
acquired from encoder 12, full-scene composition por- 
tion 16 determines whether the acquired BIFS data is a 
BIFS command other than Scene Replace (step S11). 
[0070] If the acquired BIFS data is a BIFS command 
other than Scene Replace then full-scene composition 
portion 16 composites full-scene BIFS data received 
from full-scene storage portion 18 and the BIFS com- 
mand received from encoder 12 together to generate 
new, full-scene BIFS data (step S13). 
[0071] For example, if full-scene BIFS data is repre- 
sented by a scene graph configured of a node "a square 
frame" (a still picture), a node "a news caster" (an ani- 
mated image) and a node "a Japanese speech 0 and a 
BIFS command is input instructing that a node "an illus- 
tration" (a CG image) be added (Insertion) then new full- 
scene BIFS data is generated with this new node added 
thereto. 

[0072] Then this new full-scene BIFS data is stored in 



full-scene storage portion 18 (step S14), when full- 
scene BIFS data having been stored in full-scene stor- 
age portion 18 is overwritten and thus updated to the 
new full-scene BIFS data. 
5 [0073] At step S1 1 when full-scene composition por- 
tion 16 determines that the Scene Replace BIFS com- 
mand is received, new full-scene BIFS data is to be 
transmitted. Thus, full-scene BIFS data read from full- 
scene storage portion 1 8 and a Bl FS command received 
to from encoder 12 are not composited together and the 
new full-scene BIFS data is stored in full-scene storage 
portion 18. Note that if in a data stream, full-scene BIFS 
data is initially input, the full-scene BIFS data is of 
course stored intact in full-scene storage portion 18. 
is [0074] Thus, full-scene storage portion 1 8 constantly 
stores the latest full-scene BIFS data. 
[0075] Then as shown in Fig. 8 if there occurs an oc- 
casion to transmit full-scene BIFS data stored in full- 
scene storage portion 18 then output switch portion 19 
20 switches its multiplexer 14 connection to terminal Y 
(step S21) to output to the multiplexer 14 the full-scene 
BIFS data stored in full-scene storage portion 18. Mul- 
tiplexer 14 multiplexes received full-scene BIFS data 
and each object data together and thus outputs the 
2S same to transmission and reception portion 15, which 
in turn transmits the full-scene BIFS data to reception 
system 20 (step S22). When the transmission has com- 
pleted, output switch portion 19 switches its connection 
back to terminal X (step S23). 
30 [0076] Full-scene BIFS data may be transmitted as 
described above for example if reception system 20 is- 
sues a request to transmit full-scene BIFS data, or it may 
be transmitted periodically whenever a predetermined 
period of time elapses whether reception system 20 is- 
35 sues such a request or not. Note that if it is transmitted 
regularly whenever a predetermined period of time 
elapses, a timer (not shown) is provided and when the 
timer counts a predetermined period of time f output 
switch portion 1 9 responsively switches to terminal Y to 
40 transmit full-scene BIFS data stored in full-scene stor- 
age portion 18. 

[0077] Reception system 20 operates as described 
below: initially, its normal operation will be described in- 
cluding requesting transmission system 10 to transmit 
45 full-scene BIFS data. 

[0078] Transmission system 10 transmits a multi- 
plexed, media object stream, which is received by trans- 
mission and reception portion 24 and separated by de- 
multiplexed into each object data, an object descriptor 
so and BIFS data. 

[0079] Then each object data is decoded for example 
by decoder 40a for example via decoding buffer 30a and 
this decoding process generates a composition unit 
which is in turn stored for example in composition mem- 
55 ory 50a. 

[0080] The object descriptor is decoded by decoder 
41 via decoding buffer 31 and this decoding process 
generates a composition unit which is in turn stored in 
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composition memory 51 . 

[0081] The BIFS data is decoded by decoder 42 via 
decoding buffer 32 and this decoding process generates 
a composition unit which is in turn stored in composition 
memory 52. 

[0082] Then compositor 60 employs the method de- 
scribed with reference to Fig. 6 to composite each com- 
position unit stored in composition memory group 50 at 
a composition memory, to generate data for an output 
for example to be displayed. 

[0083] Decoder 42, BIFS check portion 70 and re- 
transmission request generation portion 80 provide a 
process, as shown in Figs. 9 and 10. 
[0084] With reference to Fig. 9, BIFS check portion 70 
determines whether BIFS data has been received via 
decoding buffer 32 (step S30) and if not then step S30 
is repeated. If BIFS data has been received via decoding 
buffer 32 then decoder 42 decodes the BIFS data (step 

531) . Decoder 42 then outputs a composition unit to 
composition memory 52. 

[0085] BIFS check portion 70 uses the composition 
unit output from decoder 42, to check BIFS data (step 

532) . Step S32 is provided according to the flow chart 
shown in Fig. 10. 

[0086] With reference to Fig. 10, BIFS check portion 
70 determines from the composition unit received from 
decoder 42 whether a packet (an SL packet) has been 
received (step S40). If so, BIFS check portion 70 starts 
an incorporated timer (step S41). 
[0087] BIFS check portion 70 then determines wheth- 
er a subsequent packet has been received (step S42). 
If it determines that a subsequent packet has been re- 
ceived, BIFS check portion 70 determines whether the 
packet that BIFS check portion 70 determined had been 
received at step S40 and the packet that BIFS check 
portion 70 determines has been received at step S42 
have their respective sequence numbers in succession 
(step S43). More specifically, BIFS check portion 70 de- 
termines whether an SL packet has a header including 
a successive packet sequence number. If so then BIFS 
check portion 70 determines that the BIFS data has no 
abnormalities (step S44). 

[0088] If at step S43 BIFS check portion 70 deter- 
mines that sequence numbers are not in succession that 
means that a packet is missing and the BIFS data con- 
tained In the missing packet is thus also missing. As 
such, BIFS check portion 70 determines that there is an 
abnormality, determining that BIFS data is missing as a 
transmission line has an abnormality or for a similarly 
reason (step S46). It also determines that there is an 
abnormality if a subsequent packet is not received for a 
predetermined period of time after the timer has been 
started (steps S45 and 546). 

[0089] Again with reference to Fig. 9, if at step S32 
BIFS check portion 70 determines that there is an ab- 
normality then the control moves from step S33 to step 
S34 and BIFS check portion 70 outputs to retransmis- 
sion request generation portion 80 the information indi- 



cating that there is an abnormality. Responsively, re- 
transmission request generation portion 80 outputs a re- 
transmission request signal to transmission and recep- 
tion portion 24 to request transmission system 10 to 

5 transmit full-scene BIFS data. Transmission and recep- 
tion portion 24 receives and transmits the retransmis- 
sion request signal to transmission system 10 (step 
S34). Note that if BIFS check portion 70 determines that 
there is no abnormality (in the Fig. 10 at step S44) the 

10 request to transmit full-scene BIFS data, as described 
above, is not issued. 

[0090] The issuance of a request from retransmission 
request generation portion 80 to transmit full-scene 
BIFS data serves as an occasion for transmission sys- 

15 tern 1 0 to transmit a full-scene BIFS and transmission 
system 1 0 transmits full-scene BIFS data stored in full- 
scene storage portion 1 8. More specifically, when re- 
ception system 20 transmits a retransmission request 
signal, transmission system 10 receives the retransmis- 

20 sion request signal at transmission and reception por- 
tion 15, which in turn outputs the received restransmis- 
sion request signal to output switch portion 19. Output 
switch portion 1 9 having received the retransmission re- 
quest signal switches the terminal for the input for mul- 

25 tiplexer 14 from terminal X to terminal Y to read full- 
scene BIFS data stored in full-scene storage portion 18 
and outputs it to multiplexer 1 4. Multiplexer 14 multiplex- 
es the full-scene BIFS data output from full-scene stor- 
age portion 1 8 and each object data together and out- 

30 puts the multiplexed data to transmission and reception 
portion 1 5. Transmission and reception portion 1 5 trans- 
mits to reception system 20 the multiplexed data re- 
ceived from multiplexer 14. 

[0091 ] Note that while in the reception system 20 op- 
35 eration as described above a determination is made that 
there is an abnormality if any packet is not received with- 
in a predetermined period of time after the timer has 
been started, a packet for BIFS data is not always trans- 
mitted from transmission system 10 periodically and 
40 BIFS check portion 70 may accordingly be adapted to 
avoid determining that there is an abnormality if any 
packet has not been received within the predetermined 
period of time, although, as shown in Fig. 10, determin- 
ing as an abnormality no reception of a packet within a 
45 predetermined period of time and requesting retransmit- 
ting the packet allows the latest full-scene BIFS data to 
be constantly obtained. 

[0092] While in the above description whether or not 
a packet has a successive sequence number is referred 

50 to to determine whether received BIFSdatais abnormal, 
the present invention is not limited thereto and whether 
or not an access unit (AU) has a successive sequence 
number may be determined and if not a determination 
that there is an abnormality may be made. 

55 [0093] In the transmission and reception system of 
the first embodiment transmission system 10 can con- 
stantly store the latest full-scene BIFS data, and trans- 
mit it to reception system 20 at regular intervals and 
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transmit it in response to a request received from recep- 
tion system 20. As such, if a fault occurs in a transmis- 
sion line between transmission system 1 0 and reception 
system 20 and BIFS data such as a BIFS command is 
missing (or delayed), reception system 20 still can ob- 5 
tain the latest full-scene BIFS data and use it to output 
a normal scene. 



Second Embodiment 



w 



[0094] A second embodiment provides a reception 
system 1 000 and a transmission system 1 500 that have 
a relationship as will be described with reference to Fig. 
1 1 . Transmission system 1 500 delivers to reception sys- 
tem 1 000 object data such as a speech and an animated is 
image, scene description data describing an objects ar- 
rangement and relationship In time (BIFS data), and ob- 
ject descriptor OD associating a constituent of a scene 
with an object. Each stream is assigned an elementaryTl 
stream identification (ES-ID) provided to identify the |Uo 
stream. Each stream is encoded in a respective system. J 
[0095] A scene description is represented by a scene 
graph using a scene description language BIFS. At a 
node referring to an object an object descriptor identifi- 
cation (OD-ID) is designated for identifying an object. 2s 
Object descriptor OD refers to object-descriptor identi- 
fication OD-ID and elementary-stream identification ES- 
ID to associate a scene description and an object with 
each other. 

[0096] In the second embodiment reception system 30 
1000 displays a scene from such encoded data. Note, 
however, that if a portion determining whether data is 
missing 6 determines that BIFS data has not yet arrived 
or it is missing (or delayed) then default BIFS data stored 
in a default-BIFS storage portion 8 is used to display a 35 
scene. 

[0097] One example of reception system 1 000 of the 
second embodiment will now be described with refer- 
ence to Fig. 12. The Fig. 12 reception system 1 000 sys- 
tem includes a demultiplexer 1 developing a multiplexed 40 
stream into individual streams, decoding buffers 2A-2D 
holding data output from demultiplexer 1 , decoders 3A- 
3D decoding data of decoding buffers 2A-2D, respec- 
tively, composition memories 4A-4D holding outputs re- 
spectively output from decoders 3A-3D, respectively, 45 
and a compositors compositing a composition unit CU.' 
[0098] Demultiplexer 1 outputs access unit AU corre- 
sponding to a processing unit for the time management 
for decoding and composition, synchronization and the 
like. Transmission system 1500 transmits access unit so 
AU in the form of a SL packet, one at a time (or divides 
access unit AU into a plurality of SL packets for trans- 
mission). 

[0099] The SL packet has a header with time informa- 
tion added thereto for synchronization management. 55 
The time information includes a decoding-time stamp 
DTS and a composition-time stamp CTS. Decoding- 
time stamp DTS is used for time management for a de- 



coding process. Composition-time stamp CTS is used 
for time management for a composition process. 
[0100] The SL packet's header has also a number Ja~) 
packet sequence number) added thereto. The packet I 
sequence number has a successive value for each SL / 
packet. J 
[0101] Note that access unit AU can also have a suc- 
cessive number (AU sequence number) chronologically 
added thereto. 

[0102] For each object there exist a decoder and a 
buffer, In a one to one correspondence. For example, 
data for an animated image is decoded by decoder 3A 
and speech data is decoded by decoder 3B. Further- 
more, object descriptor OD is decoded by decoder 3C 
and BIFS data is decoded by decoder 3D. 
[0103] Reception system 1000 further includes por- 
tion determining whether BIFS data is missing 6, an OD 
change portion 7 changing object descriptor OD, and 
default-BIFS storage portion 8 holding default BIFS da- 
ta. 

[0104] If BIFS data is missing, the portion determining 
whether data is missing 6 reads default BIFS data (de- 
coded data) from default-BIFS storage portion 8 and 
supplies the read data to composition memory 4D. 
[0105] OD change portion 7, in response to a deter- 
mination made by the portion determining whether Bl FS 
data is missing 6, changes an output of decoder 3C ac- 
cording to default BIFS data. Thus a correspondence is 
provided between elementary-stream identification ES- 
ID in object data and object-descriptor identification OD- 
ID of default BIFS data. 

[0106] With reference to Fig. 13A, normally, BIFS data 
B1 is used to composite together an object CB1 for an 
animated image having a geometry A constituting a 
scene S 1 and an object OB2 for an animated image 
having a geometry B constituting scene S1 and thus dis- 
play scene 1 . BIFS data B2 is then used to composite 
together object OB 1 for the animated image having ge- 
ometry A constituting a scene S2 and object OB2 for the 
animated image having geometry B constituting scene 
S2 and thus display scene S2. 

[0107] With reference to Fig. 13, if BIFS data is miss- 
ing (a fault has occurred), default BIFS data B1 # stored 
in default-BIFS storage portion 8 is used to composite 
together object OB 1 for the animated image having ge- 
ometry A constituting scene S 1 and object OB2 for the 
animated image having geometry B constituting scene 
S1 . Then, default BIFS data B2# stored in default-BIFS 
storage portion 8 is used to composite together object 
OB1 for the animated image having geometry A consti- 
tuting scene S2 and object OB2 for the animated image 
having geometry B constituting scene S2. 
[0108] Reference will now be made to Figs. 14 and 
15 to describe a flow of acquiring session data in the 
MPEG-4 system. Fig. 14 corresponds to a time when a 
session starts and Fig. 15 corresponds to a time when 
a scene is changed. 

[01 09] With reference to Fig. 1 4, at step S1 00 an initial 
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object descriptor is acquired. Thereafter, reception is 
waited. Then the control moves to a process for chang- 
ing a scene. 

[0110] The initial object descriptor is data that is ini- 
tially transmitted in a MPEG-4 session. Acquisition of 
the initial object descriptor reveals a setting of a header 
of an SL-packetized BIFS stream and that of a header 
of an SL-packetized object-descriptor stream. 
[0111] With reference to Fig. 15, when a scene is to 
be changed, at step S110 an object-descriptor stream 
is acquired. At step S111 a BIFS command is acquired 
to update a constituent of a scene. 
[0112] Acquisition of a BIFS stream and an object-de- 
scriptor stream reveals a scene configuration, the type 
and number of a stream/streams, and a setting of a 
header of an SL packet. 

[01 1 3] Then at step S1 1 2 a video stream and an audio 
stream are acquired. After all streams are acquired, re- 
ception is waited. 

[0114] Corresponding to such a flow, reception sys- 
tem 1000 provides a process as shown in Fig. 16. Fig. 
1 6 represents a flow of a process in reception system 
1000 in changing a scene (including displaying an initial 
scone). 

[01 1 5] With reference to Fig. 1 6, initially at step S1 20 
an object-descriptor stream is acquired. Subsequently 
at step S121 the portion determining whether data is 
missing 6 determines whether BIFS data has arrived, L 
e. , whether it is missing or not. If BIFS data is determined 
to have arrived (or it is not missing) then the control goes 
to step S123 and if BIFS data is not determined to have 
arrived or it is missing then the control goes to step 
S122. 

[01 1 6] At step S1 22, the portion determining whether 
data is missing 6 exerts control to supply composition 
memory 4D with default BIFS data stored in default- 
BIFS storage portion 8. Controlled by the portion deter- 
mining whether data is missing 6, OD change portion 7 
changes an object descriptor to allow it to be applied to 
BIFS data. The control then goes to step S123. 
[0117] At step S123 a video stream and an audio 
stream are acquired. Reception is then waited. 
[0118] As such, if BIFS data is missing then default 
BIFS data is used to temporarily display a scene. There- 
after when a formal BIFS data has arrived it is used to 
display a scene. 

[0119] An example of a process for determining 
whether a BIFS has arrived will now be described with 
reference to Fig. 17. Initially at step S130 an SL packet 
(or an access unit BIFS AU corresponding to BIFS data) 
is received. Then at step S131 a timer included in the 
portion determining whether BIFS data is missing 6 is 
started and counts time. 

[0120] At step S132 a determination is made as to 
whether a subsequent SL packet (or access unit BIFS 
AU) has been received. If not then the control goes to 
step S133 and if so then the control goes to step S134. 
[0121] At step S133 the time counted by the timer is 



referred to to determine whether a predetermined period 
of time has elapsed since the timer was started (i.e., 
whether the timer has expired). If not then the control 
goes to step S132. If the predetermined period of time 
5 has elapsed, then a determination is made that BIFS 
data has not yet arrived (or it is missing). 
[0122] At step S134 a determination is made as to 
whether two SL packets (or access units BIFS AUs) 
have their respective headers with their respective set 
10 packet sequence numbers (or AU sequence numbers) 
in succession. If so then a determination is made that 
BIFS data are not missing and thus successively arrive. 
If not then a determination is made that BIFS data has 
not yet arrived (or it is missing). 
15 [01 23] Thus if a subsequent BIFS data does not arrive 
within a predetermined period of time or sequence num- 
bers are not in succession then a determination is made 
that BIFS data Is missing (or delayed). 
[0124] Note that if the fault-BIFS storage portion 8 is 
20 adapted to store an output of decoder 3D, BIFS data 
used before a missing occurs can be used as default 
BIFS data. This is particularly effective for a scene con- 
figuration which does not vary significantly. 
[0125] In the second embodiment, as shown in Fig. 2, 
25 reception system 1000 includes default BIFS storage 
portion 8 holding default BIFS data. The present inven- 
tion, however, is not limited thereto and if BIFS data is 
missing or delayed, rather than the default BIFS data 
held in default-BIFS storage portion 8 the BIFS data that 
30 compositors 5 immediately previously used to describe 
a scene may be used to describe a scene. 
[0126] Furthermore, in normal operation, decoding 
buffer 2D outputs BIFS data to decoder 3D and also out- 
puts BIFS data to default-BIFS storage portion 8 and 
35 stores it therein. When the portion determining whether 
data is missing 6 detects that BIFS data is missing (or 
delayed) default-BIFS storage portion 8 uses its stored 
BIFS data to generate BIFS data analogous to BIFS da- 
ta provided for an element having already been dis- 
40 played in a scene. Default-BIFS storage portion 8 then 
outputs the generated BIFS data to decoder 3D. There- 
after, a normal route is followed to input the BIFS data 
to compositors 5 and use it to display a scene. 
[0127] Thus a configuration allowing BIFS data to be 
45 automatically generated may be adopted. 

[0128] In the second embodiment, if BIFS data corre- 
sponding to received object data does not arrive (if it is 
missing or delayed), default scene description informa- 
tion can be used to display a scene. Thus a scene can 
so be displayed with high quality, without failing to display 
a corresponding object, delaying displaying the same, 
or displaying a content with contradiction. 



55 



Third Embodiment 

[0129] A third embodiment provides a multimedia sys- 
tem, as described with reference to Figs. 18 and 19. In 
the third embodiment, as shown in Fig. 18, a reception 



10 



19 



EP1 133 189 A2 



20 



system (a receiving side) 2000 is provided with a mem- 
ory 90 for accumulating BIFS data and object descrip- 
tors. If movie software or the like is used to display a 
cumulative medium, a transmission system (a transmit- 
ting side) 2500 initially transmits a BIFS stream and an 
object-descriptor stream via a network NE. These data 
transmitted are accumulated in memory 90. Thereafter 
transmission system 2500 transmits an object stream. 
[01 30] With reference to Fig. 1 9A, if the reception sys- 
tem is of a typical MPEG-4 system, BIFS data B 1 trans- 
mitted simultaneously with an object is used to compos- 
ite together object OB1 for the animated image having 
geometry A constituting scene S1 and object OB2 for 
the animated image having geometry B constituting 
scene S1 . Similarly, BIFS data B2 is used to composite 
together object OB1 for an animated image having ge- 
ometry A constituting scene S2 and object OB2 for the 
animated Image having geometry B constituting scene 
S2. 

[0131] Correspondingly, reception system 2000, as 
shown in Fig. 19B, initially accumulates BIFS data in 
memory 90, and then provides a composition of re- 
ceived objects OB 1 and OB2 for animated images that 
is based on the corresponding BIFS data extracted from 
memory 90, and displays scene S1 . Object data and 
BIFS data are synchronized with each other, based on 
a time stamp (decoding-time stamp DTS and composi- 
tion-time stamp CTS). 

[0132] Similarly for scene S2 received objects OB1 
and OB2 for animated images are composited together 
based on the corresponding BIFS data extracted from 
memory 90. 

[0133] Reference will now be made to Figs. 20 and 
21 to describe a flow of acquiring session data in recep- 
tion system 2000. Fig. 20 represents a process when a 
session starts and Fig. 21 represents a process for ex- 
tracting a BIFS, both by way of example. 
[0134] With reference to Fig. 20, initially at step S200 
an initial object descriptor is acquired. Thereafter the 
control moves to the Fig. 20 BIFS extraction process. 
[0135] With reference to Fig. 21 , initially at step S21 0 
an object-descriptor stream is extracted from memory 
90 and thus acquired. Then at step S211 BIFS data is 
extracted and thus acquired. 

[0136] Data stored In memory 90 are extracted ac- 
cording to a time stamp successively as an initial object 
descriptor arrives. Arrival of corresponding object data 
is waited. 

[0137] Then at step S21 2 a video stream and an audio 
stream are acquired. 

[0138] When an object data has arrived the control 
moves to step S213. 

[0139] Step S213 provides an association between 
data that is required for displaying a scene. More spe- 
cifically, object descriptor OD, BIFS data, and object da- 
ta that has arrived (and other object data to be required) 
are synchronized with one another according to a time 
stamp. After step S213 the control goes to a decoding 



process. 

[01 40] To achieve the synchronization according to a 
time stamp, the time stamp is attached to an SL packet 
in transmission system 2500 according to a predeter- 
5 mined rule. For example, "the video, audio, BIFS and 
object descriptor OD that are to be initially decoded have 
the same time stamp" and from this information a com- 
mon origin of time axes is determined and a correspond- 
ence between video, audio, BIFS and object descriptor 
10 00 are thus uniquely determined. 

[0141] Reference will now be made to Fig. 22 to de- 
scribe one example of reception system 2000 in the sec- 
ond embodiment. A reception system 2000A shown in 
Fig. 22 includes demultiplexer 1, decoding buffers 2A- 
15 2D, decoders 3A-3D, composition memories 4A-4D, 
compositor 5, a memory 90 accumulating BIFS data and 
object descriptor OD (a BIFS and OD accumulation por- 
tion 90), an initlal-OD determination portion 21 deter- 
mining an initial object descriptor, and a time stamp ad- 
20 justment portion 22. 

[0142] Initial-OD determination portion 21 determines 
whether an initial object descriptor has arrived. If an in- 
itial object descriptor has arrived then data accumulated 
in BIFS and OD accumulation portion 90 are succes- 
2s sively extracted according to a time stamp. The extract- 
ed data are successively output to time stamp adjust- 
ment portion 22. 

[0143] Time stamp adjustment portion 22 refers to a 
time stamp of an object (video and audio) output from 
30 demultiplexer 1 , to output BIFS data and object descrip- 
tor OD that have the same time stamp, which are in turn 
received by decoders 3C and 3D, respectively. Thus an 
object is composited according to the time stamp. 
[0144] Reference will now be made to Fig. 23 to de- 
35 scribe another example of reception system 2000 in the 
third embodiment. A reception system 2000B shown in 
Fig. 23 includes demultiplexer 1, decoding buffers 2A- 
2D, decoders 3A-3D, composition memories 4A-4D, 
compositor 5, BIFS and OD accumulation portion 90, 
^0 initial-OD determination portion 21, and a time stamp 
adjustment portion 23. 

[0145] In reception system 2000B BIFS data and ob- 
ject descriptor OD that are initially transmitted are de- 
coded by decoders 3C and 3D, respectively, and a result 
43 of the decoding is accumulated In BIFS and OD accu- 
mulation portion 90. 

[0146] Controlled by initial-OD determination portion 
21, data accumulated in BIFS and OD accumulation 
portion 90 are extracted successively according to a 
so time stamp and thus output to time stamp adjustment 
portion 23. 

[0147] Time stamp adjustment portion 23 refers to a 
time stamp of an object (video and audio) output from 
demultiplexer 1 , to output BIFS data and object descrip- 
55 tor OD that have the same time stamp, which are in turn 
received by composition memories 4C and 4D, respec- 
tively Thus an object is composited according to the 
time stamp. 
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[0148] Reception systems 2000A and 2000B both 
communicate BIFS data and an object descriptor ac- 
cording to a protocol that has a retransmission function 
such as TCP and is thus highly reliable (i.e., free from 
missing data). Using such a highly reliable protocol can 
eliminate the problem of missing and delayed BIFS da- 
ta. Furthermore, these data that are transmitted are sim- 
ply stored rather than used in a composition process and 
it is thus not disadvantageous if a network is delayed. 

Fourth Embodiment 

[01 49] A fourth embodiment provides a reception sys- 
tem (a receiving side) 3000 and a transmission system 
(a transmitting side) 21 00 outlined with reference to Fig. 
24. Transmission system 21 00 delivers to reception sys- 
tem 3000 object data for a speech, an animated Image 
and the like, scene description data describing an ob- 
ject's arrangement and relationship in time (BIFS data) 
and object descriptor OD associating a constituent of a 
scene with an object. Transmission system 2100 deliv- 
ers the data and object descriptor OD to reception sys- 
tem 3000 in a stream. These data have a time stamp 
added thereto for time management. Each stream is de- 
coded in a respective system. 

[01 50] As has been described previously, a scene de- 
scription is represented by a scene graph employing a 
scene description language BIFS. 
[01 51 ] Reception system 3000 has a Bl FS check por- 
tion 110 checking BIFS data and a pass/fail determina- 
tion database 120. Reception system 3000 determines 
whether there is a contradiction between scene descrip- 
tions that is attributed to missing (or delayed) BIFS data 
and reception system 3000 accordingly provides a proc- 
ess to reduce a contradiction in a content for display. 
[0152] Reception system 3000 has a function to 
check BIFS data, as described with reference to Fig. 25. 
With reference to the figure, initially at step S1000 BIFS 
command is received and decoded. 
[01 53] At step S1 01 pass/fail determination database 
120 receives an inquiry. At step S102 reference infor- 
mation for determination contained in pass/fail determi- 
nation database 120 is referred to to examine whether 
a BIFS command has a contradiction (i.e., whether a 
scene description required is missing between a scene 
description being displayed and the BIFS command of 
interest). If so then at step S103 a process addressing 
the contradiction is executed. If not or the step S103 
process completes then a subsequent process follows. 
[0154] As the process addressing and thus reducing 
a contradiction in displaying (step S103), when a con- 
tradiction is introduced the scene description with the 
detected contradiction (that is to be composited and dis- 
played) may be discarded, which will be referred to as 
a "process A" hereinafter for the sake of convenience, 
or a scene description being displayed and also contra- 
dicting a scene description to be composited may be 
modified, which will be referred to as a "process B" here- 



inafter for the sake of convenience. 
[0155] A description will now be made of a reception 
system having a function to modify an exact, displayed 
content to provide the post priority addressing process. 

5 A reception system 3000A shown in Fig. 26 includes de- 
multiplexer 1 receiving a multiplexed stream and devel- 
oping it into individual streams, decoding buffers 2A-2D 
holding data output from demultiplexer 1 , decoders 3A- 
3D decoding data of decoding buffers 2A-2D, respec- 

10 tively, composition memories 4A-4D holding outputs of 
decoders 3A-3D, respectively, and compositors 5 com- 
positing composition unit CU for displaying a scene. 
[0156] Demultiplexer 1 outputs access unit AU serv- 
ing as a processing unit for time management for de- 

15 coding and composition, synchronization and the like. 
Transmission system 2100 transmits access unit AU in 
the form of an SL packet, one at a time (or divides ac- 
cess unit AU into a plurality of SL packets for transmis- 
sion). 

20 [0157] For each object there exist a decoder and a 
buffer, in a one to one correspondence. For example, 
data for an animated image is decoded by decoder 3A 
and speech data is decoded by decoder 3B. Further- 
more, object descriptor OD is decoded by decoder 3C 

25 and BIFS data is decoded by decoder 3D. 

[0158] Reception system 3000 also includes BIFS 
check portion 110 checking BIFS data, pass/fail deter- 
mination database 1 20 storing reference information for 
determination that BIFS check portion 1 1 0 uses to make 

30 a pass/fail determination, and a portion 12A modifying 
a displaying condition, modifying an exact content dis- 
played. 

[0159] BIFS check portion 110 refers to a displaying 
condition in compositor 5 to check an output of decoder 
35 3D, which will be referred to as scene description B1 . 
More specifically, pass/fail determination database 120 
is referred to to determine whether there is a contradic- 
tion introduced between scene description B0 being dis- 
played and scene description B1 received. 
40 [0160] If a determination is made that there is a con- 
tradiction introduced that is attributed to missing or de- 
layed BIFS data, the portion 12A modifying a displaying 
condition provides a process to change a content in 
compositor 5 that is currently displayed. More specifi- 
cs cally, it directly modifies data contained In compositor 5 
or being developed on a memory contained in a terminal 
connected to compositor 5. Thus, a scene can be 
changed without contradiction. 

[0161] The fourth embodiment provides another ex- 
50 emplary reception system, as described with reference 
to Fig. 27. A reception system 3000B shown in Fig, 27 
provides process B. functioning to internally produce a 
BIFS command for modifying a scene description being 
displayed. 

55 [0162] Reception system 3000B includes demulti- 
plexer 1, decoding buffers 2A-2D, decoders 3A-3D, 
composition memories 4A-4D, compositor 5, BIFS 
check portion 110, pass/fail determination database 12, 
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a portion 12B modifying a displaying condition, and a 
BIFS command generation portion 130. 
[01 63] As has been described above, Bl FS check por- 
tion 110 refers to pass/fail determination database 120 
to determine whether there is a contradiction introduced 
between scene description BO being displayed and 
scene description B1 received. If BIFS check portion 
1 1 0 determines that there is a contradiction in BIFS data 
then the portion 12B modifying a displaying condition 
instructs BIFS command generation portion 130 to gen- 
erate a command for changing the current displaying 
condition. 

[0164] Controlled by the portion 12B modifying a dis- 
playing condition, BIFS command generation portion 
130 generates a BIFS command for changing a scene 
description. Thus a BIFS command required for modi- 
fying the current displaying condition Is generated. 
[0165] BIFS command generation portion 130 pro- 
vides an output which is in turn fed to decoding buffer 
2D, follows the same route as BIFS data received from 
the transmitting side, and it is processed. According to 
the BIFS command generated a scene graph indicating 
a current display is changed. If the generated BIFS com- 
mand and the corresponding object data need to be as- 
sociated with each other, object descriptor OD is updat- 
ed according to the BIFS command internally generat- 
ed. Thus, a scene being displayed is modified according 
to the internally generated BIFS command. 
[0166] After the modification, a scene is displayed 
based on scene description B1 having been checked by 
BIFS check portion 110. Thus a scene can be changed 
without contradiction. 

[0167] The fourth embodiment provides still another 
example of the reception system, as described with ref- 
erence to Fig. 28. A reception system 3000C shown in 
Fig. 28 provides process B, functioning to request trans- 
mission system 2100 to retransmit a BIFS command. 
[0168] Reception system 3000C includes demulti- 
plexer 1, decoding buffers 2A-2D : decoders 3A-3D, 
composition memories 4A-4D, compositor 5 : BIFS 
check portion 110, pass/fail determination database 
1 20, a portion 1 2C modifying a displaying condition , and 
a retransmission request generation portion 140. 
[0169] As has been described above, BIFS check por- 
tion 110 refers to pass/fail determination database 120 
to determine whether there is a contradiction introduced 
between scene description B0 being displayed and 
scene description B 1 received. If there is a contradiction 
in BIFS data, the portion 12C modifying a displaying 
condition instructs retransmission request generation 
portion 140 to request retransmitting a BIFS command. 
[0170] Controlled by the portion 12C modifying a dis- 
playing condition, retransmission request generation 
portion 140 generates data for requesting transmission 
system 2100 to retransmit a BIFS command (i.e., a re- 
transmission request). If the missing BIFS command of 
interest can be specified, retransmission request gen- 
eration portion 140 requests transmission system 2100 



to retransmit the exact missing command. If the missing 
BIFS command cannot be specified, then retransmis- 
sion request generation portion 140 requests transmis- 
sion system 2100 to go back in time, as appropriate, to 
* retransmit previous BIFS data. In response to the re- 
transmission request, transmission system 2100 trans- 
mits BIFS data. 

[0171] Thus the fourth embodiment provides a recep- 
tion system capable of reducing a contradiction in a dis- 
'0 play if there is a contradiction introduced between a 
scene being displayed and a scene description to be dis- 
played, i.e., if BIFS data is missing (or delayed). 
[01 72] A specific example thereof is shown in Fig. 29 . 
In the figure, by way of example, scene description B0 
is currently being displayed is represented by a scene 
graph formed of a node A1 (a "square frame"), a node 
A10 (a "news caster (an animated image)") and a node 
All (a "Japanese speech"), and a subsequent BIFS data 
is a BIFS command instructing that a node A12 (an 
20 "English speech") be connected to node A1 . 

[0173] If the reception system does not have a BIFS 
check function then a new scene description B1 is rep- 
resented by a scene graph formed of node A1 , node 
A10 ; node A11 ("Japanese speech") and node A12 
25 ("English speech"). 

[0174] In contrast the reception system of the fourth 
embodiment sends an inquiry to pass/fail decision da- 
tabase 120 when BIFS data has arrived. 
[0175] In response to the inquiry, a determination is 
30 for example made that if the "English speech" is con- 
nected a command deleting the "Japanese speech" is 
required, i.e., there is a contradiction introduced be- 
tween scene descriptions and a BIFS command delet- 
ing the "Japanese speech" should be required. 
35 [0176] When such a BIFS command has been detect- 
ed to be missing, reception system 3000A directly mod- 
ifies the Japanese speech current displayed. 
[01 77] Reception system 3000B internally generates 
a Bl FS command to disconnect node A1 1 from node A1 . 
40 [0178] Reception system 3000C requests retransmis- 
sion of a BIFS command for disconnecting node A11 
from node A1. 

[0179] Thus a Japanese speech currently displayed 
is terminated and it is switched to an English speech. 

45 Thus an output through the English speech is provided. 
[0180] Although the present Invention has been de- 
scribed and illustrated in detail, it is clearly understood 
that the same is by way of illustration and example only 
and is not to be taken by way of limitation, the spirit and 

50 scope of the present invention being limited only by the 
terms of the appended claims. 



Claims 



55 



Reception system (20, 1000, 1000A, 1000B, 
1000C, 3000, 3000A, 30003, 3000C) using scene 
description information describing a scene trans- 
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mitted from a transmission system (1 0, 1500, 2000, 
2500) and object data constituting said scene, to 
display said scene, comprising: 

a composition process portion (5, 4A-4D, 60, 
50a-50n, 51, 52) using said scene description 
information to composite a corresponding said 
object data to display said scene; and 
a display control portion (6, 8, 12A, 22, 70, 80, 
90, 110, 120) controlling said composition proc- 
ess portion to resolve an error in said scene dis- 
played attributed to non-arrival of scene de- 
scription information provided to describe said 
scene. 

2. The reception system of claim 1 , wherein in control- 
ling said composition process portion (5, 4A-4D, 60, 
50a-50n, 51, 52) to resolve said error said display 
control portion (6, 8, 12A, 22, 70, 80, 90, 110, 120) 
supplies said composition process portion (5, 4A- 
4D, 60, 50a-50n, 51 , 52) with said scene description 
information for resolving said error and controls said 
composition process portion (5, 4A-4D, 60, 50a- 
50n, 51 , 52) to use said scene description informa- 
tion for resolving said error to resolve said error. 

3. The reception system of claim 2, wherein said dis- 
play control portion (6, 8, 12A, 22, 70, 80, 90, 110, 
120) includes: 

a determination portion (6) determining arrival/ 
non-arrival of said scene description informa- 
tion; and 

a control portion (6) supplying said composition 
process portion (5, 4A-4D, 60, 50a-50n, 51 , 52) 
with default scene description information and 
controlling said composition process portion (5, 
4A-4D, 60, 50a-50n, 51 , 52) to use said default 
scene description information to execute said 
composition when said determination portion 
(6) determines that said scene description in- 
formation has not yet arrived. 



10 6. 



data; and 

said control portion (6) also includes a modifi- 
cation portion (7) operative in response to a de- 
termination of said determination portion to use 
said default scene description information to 
modify corresponding said scene control infor- 
mation for output to said composition process 
portion (5, 4A-4D, 60, 50a-50n, 51 , 52). 

The reception system of claim 4, wherein said mem- 
ory (8) stores said scene description information 
previously received. 



7. The reception system of claim 2, said scene de- 
15 scription information including full-scene descrip- 
tion information for describing a full scene and a 
scene description command partially changing said 
scene, wherein said display control portion (6, 8, 
12A, 22, 70, 80, 90, 110, 120) includes: 

20 

a determination portion (70) determining 
whether received scene description informa- 
tion has an abnormality; and 
a transmission requesting portion (80) request- 
25 ing said transmission system to transmit full- 

scene description information when said deter- 
mination portion (70) determines that received 
scene description information has an abnor- 
mality. 

30 

8. The reception system of claim 7, wherein when said 
scene description information does not have a suc- 
cessive sequence number said determination por- 
tion (70) determines that there is an abnormality. 

35 

9. The reception system of claim 8, wherein said se- 
quence number is a sequence number of a sink- 
layer packet. 

40 10. The reception system of claim 1 , wherein said dis- 
play control portion (6, 8, 12A, 22, 70, 80, 90, 110, 
120) includes: 



The reception system of claim 3, wherein said con- 
trol portion (6) includes a memory (8) storing said «5 
default scene description Information read accord- 
ing to a determination made by said determination 
portion (6). 



5. The reception system of claim 4, wherein: 



50 



said transmission system also transmits scene 
control information associating said object data 
and said scene description information with 
each other; 55 
said composition process portion includes a 
compositor (5) using said scene control infor- 
mation to composite corresponding said object 



a check portion (110, 120) checking whether 
there is a contradiction between received 
scene description information and a current dis- 
play; and 

a control portion (12A-12C) eliminating a con- 
tradiction in a display occurring in said compo- 
sition process portion (5, 4A-4D, 60, 50a-50n, 
51 , 52) when said check portion (110, 120) de- 
termines that said contradiction has been intro- 
duced. 

11. The reception system of claim 10, wherein said 
check portion (110, 120) includes a pass/fail deter- 
mination database (120) storing information re- 
ferred to to make said determination. 
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12. The reception system of claim 11, wherein said 
check portion (110, 120) refers to said information 
referred to to make said determination and if it is 
found therefrom that scene description information 
to be received is missing or delayed then said check 
portion (110, 120) determines that said contradic- 
tion has been introduced. 

13. The reception system of claim 12, wherein said con- 
trol portion (12A-12C) is operative in response to 
an output of said check portion ( 1 1 0, 1 20) to modify 
said current display. 

14. The reception system of claim 10, wherein said con- 
trol portion (12A-12C) is operative in response to 
an output of said check portion (110, 120) to modify 
scene description information being used for said 
current display. 
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A transmission system transmitting scene descrip- 
tion information including one of full-scene descrip- 
tion information for describing a full scene and a 
scene description command for partially changing 
said scene, and an object data constituting said 
scene, said transmission system comprising: 



20 



25 



a scene description information encoding por- 
tion (12) encoding said scene description infor- 
mation; 

an object encoding portion (10a-1 On) encoding 30 
said object data; 

a storage portion (18) storing said full-scene 
description information; 
a composition portion (16) operative in re- 
sponse to a predetermined scene description 35 
command received from said scene description 
information encoding portion (1 2), to composite 
full-scene description information stored in said 
storage portion (18) and said predetermined 
scene description command together to gener- *o 
ate and use new full-scene description informa- 
tion to overwrite and thus update said full- 
scene description information in said storage 
portion (18) for storage; 

a multiplexing portion (14) multiplexing togeth- 45 
er scene description information received from 
said scene description information encoding 
portion (12) or full-scene description informa- 
tion received from said storage portion (1 8) and 
object data received from said object encoding so 
portion (10a-1 On); and 

a switch portion (19) switching an input for said 
multiplexing portion (14) between said scene 
description information encoding portion (12) 
and said storage portion (18). 55 

16. The transmission system of claim 15, wherein said 
composition portion (1 6) provides said composition 



only when a command other than a command in- 
structing that a scene be entirely replaced is re- 
ceived. 

17. The transmission system of claim 15, periodically 
transmitting to a reception system said full-scene 
description information stored in said storage por- 
tion. 

18. The transmission systemof claim 15, recefvingfrom 
a reception system receiving said object data and 
said scene description information a request made 
to transmit said full-scene description information, 
and responsively transmitting said full-scene de- 
scription information to said reception system. 

19. A transmission and reception system comprising a 
transmission system transmitting scene description 
information for describing a scene and object data 
constituting said scene, and a reception system re- 
ceiving said scene description information and said 
object data from said transmission system and us- 
ing said scene description information to composite 
said object data to display said scone, wherein: 

said transmission system includes 
a scene description information encoding por- 
tion (12) encoding said scene description infor- 
mation, 

an object encoding portion (1 0a-1 On) encoding 
said object data, 

a storage portion (18) storing full-scene de- 
scription information for describing said scene 
entirely, 

a composition portion (16) operative in re- 
sponse to a predetermined scene description 
command received from said scene description 
information encoding portion (12), to composite 
full-scene description information stored in said 
storage portion (18) and said predetermined 
scene description command together to gener- 
ate and use new full-scene description informa- 
tion to overwrite and thus update said full- 
scene description information in said storage 
portion (18) for storage, 
a multiplexing portion (14) multiplexing togeth- 
er scene description information received from 
said scene description information encoding 
portion (12) or full-scene description informa- 
tion received from said storage portion (1 8) and 
object data received from said object encoding 
portion (10a-1 On), and 

a switch portion (1 9) switching an input for said 

multiplexing portion (14) between said scene 

description information encoding portion (12) 

and said storage portion (18); and 

said reception system includes 

a determination portion (70) determining 
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whether received scene description informa- 
tion has an abnormality, and 
a transmission requesting portion (80) request- 
ing said transmission system to transmit full- 
scene description information when said deter- s 
mination portion (70) determines that received 
scene description information has an abnor- 
mality. 
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